Method and apparatus for packet loss detection

ABSTRACT

Conventional network packet traffic loss/drop monitoring mechanisms, such as that employed for pseudowire, IP flow and tunnel traffic monitoring, do not process or diagnose the aggregate counts from both endpoints of a particular pseudowire. A packet loss and detection mechanism periodically exchanges traffic packet counts to maintain an accurate diagnosis of the pseudowire health from either endpoint. Further, the raw packet counts are analyzed to identify misrouted and lost packets, as both should be considered to assess network health and congestion. The pseudowire statistics are maintained for each pseudowire emanating from a particular edge router, providing a complete view of pseudowire traffic affecting a particular edge router. Such statistics are beneficial for problem detection, diagnosis, and for verification of throughput criteria such as those expressed in Quality of Service (QOS) terms and/or SLAs (service level agreements).

BACKGROUND

In a conventional packet switched network such as a Multi-Protocol Labelswitching (MPLS) network or an IP (Internet Protocol) network,pseudowires (PWs) enable service providers to offer traditional layer-2services such as Frame Relay, ATM, and Ethernet, among others, over apacket switched core. In doing so, such providers leverage a common MPLSor IP infrastructure for both their (Layer-2) L2 and (Layer-3) L3services. While this provides both operating and capital expense savingsfor Service Providers (SPs), it also introduces new challenges in termsof operations and management (O&M/OAM/OA&M) of these new networks. Onechallenge that hasn't been completely addressed is the O&M toolset thata Service Provider can use to monitor and troubleshoot the pseudowiresthat carry the aforementioned services over their converged core.Specifically, it is important that a service provider be able to ensurethat the Service Level Agreement (SLA) they have with the customer isbeing meet. Some technologies like Pseudowire Virtual CircuitConnectivity Verification (VCCV) and other MPLS OAM techniques, such asthat disclosed in U.S. patent application Ser. No. 11/135,253 filed May23, 2005, entitled, “SYSTEM AND METHODS FOR PROVIDING A NETWORK PATHVERIFICATION PROTOCOL” is being used to help monitor the health of thepseudowire, but these tools by themselves only provide a periodicanalysis of the pseudowire in terms of reachability. Exemplaryreferences of such usage include the following: U.S. patent applicationSer. No. 11/001,149 filed Dec. 1, 2004, entitled, “SYSTEM AND METHODSFOR DETECTING NETWORK FAILURE”, U.S. patent application Ser. No.11/086,007 filed Mar. 22, 2005, entitled, “SYSTEM AND METHODS FORIDENTIFYING NETWORK PATH PERFORMANCE”, U.S. patent application Ser. No.11/072,082 filed Mar. 4, 2005, entitled, “SYSTEM AND METHODS FOR NETWORKREACHABILITY DETECTION”, and U.S. patent application Ser. No. 11/091,058filed Mar. 28, 2005, entitled, “METHOD AND APPARATUS FOR THE CREATIONAND MAINTENANCE OF A SELF-ADJUSTING REPOSITORY OF SERVICE LEVELDIAGNOSTICS TEST POINTS FOR NETWORK BASED VPNS”. However, these toolsmay not address the need service providers have to quantify the amountof packet loss for a given pseudowire. In addition to monitoring ongoingpacket loss, a tool that could measure packet loss on a pseudowire overa fixed period of time on an ad hoc basis would allow a service providerto troubleshoot customer reported problems as they occur by isolatingthem to either the provider network or the customer network.

SUMMARY

In a conventional managed information environment, such as a serviceprovider core network interconnecting local area networks (subnetworks)and corresponding end users, pseudowires provide a virtual connectionbetween edge routers defining the edges of the core network. A pluralityof pseudowires may be established between the edge routers, thereforeproviding an infrastructure for efficient traversal of the core networkby user message traffic. The pseudowires enable a native service, suchas ATM, Frame Relay, Ethernet and others for emulation over the corenetwork operable with IP, MPLS, or L2TP3 (Layer 2 Tunneling protocolVersion 3).

A typical pseudowire, therefore, defines two endpoints at the respectiveprovider edge (PE) routers that the pseudowire interconnects.Conventional pseudowire connections, as with typical packet switchedbased connections, maintain counts of packets transmitted and received.However, conventional mechanisms maintain only counts at the particularendpoint. Further, no analysis or diagnosis of the packet counts isperformed. Accordingly, configurations herein are based, in part, on theobservation that conventional pseudowire traffic monitoring does notprocess or diagnose the aggregate counts from both endpoints of aparticular pseudowire.

Unfortunately, therefore, conventional packet traffic monitoringmechanisms maintain only transmitted and received packet counts at therouter defining a respective endpoint. No correlation of packets withthe opposed endpoint is performed. Further, no analysis of packetcounts, such as misordered packets as distinguished from lost packets,is performed. Therefore, an operator or user is unable to directlymonitor packets lost or misordered over a particular connection. In alarge core network, the number of pseudowires may be substantial;thereby presenting difficulties in identifying or isolating congestionthat appears only as a slowdown at a particular edge router.

Configurations discussed herein substantially overcome the aboveshortcomings presented by conventional packet traffic monitoring byperiodically exchanging traffic packet counts to maintain an accuratediagnosis of the pseudowire health from either endpoint. Further, theraw packet counts are analyzed to identify misrouted and lost packets,as both are considered to assess network health and congestion. Thepseudowire statistics are maintained for each pseudowire emanating froma particular edge router, providing a complete view of pseudowiretraffic affecting the edge router. Such statistics are beneficial forproblem detection, diagnosis, and for verification of throughputcriteria, such as those criteria expressed in Quality of Service (QOS)terms and/or SLAs (service level agreements).

In further detail, the method of tracking packet loss includesselectively identifying a router interconnection, such as a pseudowire,IP flow or tunnel, for packet anomaly detection, such as lost andmisordered packets, and tracking the detected packet anomalies at bothtermination points of the router interconnection. The routersperiodically exchange packet anomaly statistics to synchronize packetanomaly counts between the termination points of the routerinterconnection. The anomaly statistics correspond to the terminationpoint and a particular opposed termination point. The counters,therefore, are particular PE routers having counters at pseudowiretermination points, in which the pseudowire is defined by a set oflabeled paths between provider edge routers and operable as a virtualcircuit through the core network.

Tracking the packet anomalies includes maintaining a packet loss countand a count of misordered packets. The PE routers perform the trackingby writing a sequence number in a packet. The sequence numbercorresponds to a set of packets sent on a particular pseudowire (i.e. alarger message sent as a series of packets). The opposed router receivesthe sequence number in the packet at the termination end of thepseudowire, and compares the sequence numbers to previous sequencenumbers to identify order and omission of packets. The receiving PErouter, therefore, inspects each packet to determine if the packet is inorder, preceding a previously receive packet, or advanced from a nextexpected packet. If the packet is advanced, the router computes thedifference of the received and expected packet numbers, and incrementsthe lost packets by the computed difference. If the packet is preceding(i.e. sequence number preceding one already received), the routerincrements the out of order packets.

The routers exchange packet anomaly statistics by posting (transmitting)statistics to the remote PE router by at least one of: a VCCV status,L2TP WAN error notification, Bi-Directional Forwarding Detection (BFD)keep alive, or a Label Distribution Protocol (LDP) status message. Inthe exemplary configuration, each pseudowire router maintains countspertaining to both termination points, and exchanges VCCV messagesbetween each of the termination points to propagate consistent counts ofthe lost packets and misordered packets. Tracking is selectively enabledfor particular pseudowires by initiating counts at a termination point(PE router) of a pseudowire, and transmitting the counts in a controlmessage to the opposed termination point of the pseudowire. The opposed(far end) PE router commences, responsive to the received controlmessage, the counts at the opposed termination point.

In particular configurations, counts are maintained at a plurality ofsuccessive endpoints, in which the successive endpoints are operable forconcatenation into a larger virtual circuit, each of the pairs ofsuccessive endpoints defining a particular provider subnetwork operableby a particular independent provider, such as in consecutive serviceprovider networks. In particular implementations, particularly thosehaving many PE routers, propagating consistent counts may furtherinclude aggregating the counts for each of the pseudowires. The routersdenote packets indicative of count information via a control fieldoperable to indicate control information. Such a control message maytake the form of a distinguishing control word in the message, which maybe scrutinized to initiate counts as described herein.

Alternate configurations of the invention include a multiprogramming ormultiprocessing computerized device such as a workstation, handheld orlaptop computer or dedicated computing device or the like configuredwith software and/or circuitry (e.g., a processor as summarized above)to process any or all of the method operations disclosed herein asembodiments of the invention. Still other embodiments of the inventioninclude software programs such as a Java Virtual Machine and/or anoperating system that can operate alone or in conjunction with eachother with a multiprocessing computerized device to perform the methodembodiment steps and operations summarized above and disclosed in detailbelow. One such embodiment comprises a computer program product that hasa computer-readable medium including computer program logic encodedthereon that, when performed in a multiprocessing computerized devicehaving a coupling of a memory and a processor, programs the processor toperform the operations disclosed herein as embodiments of the inventionto carry out data access requests. Such arrangements of the inventionare typically provided as software, code and/or other data (e.g., datastructures) arranged or encoded on a computer readable medium such as anoptical medium (e.g., CD-ROM), floppy or hard disk or other medium suchas firmware or microcode in one or more ROM or RAM or PROM chips, fieldprogrammable gate arrays (FPGAs) or as an Application SpecificIntegrated Circuit (ASIC). The software or firmware or other suchconfigurations can be installed onto the computerized device (e.g.,during operating system for execution environment installation) to causethe computerized device to perform the techniques explained herein asembodiments of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other objects, features and advantages of theinvention will be apparent from the following description of particularembodiments of the invention, as illustrated in the accompanyingdrawings in which like reference characters refer to the same partsthroughout the different views. The drawings are not necessarily toscale, emphasis instead being placed upon illustrating the principles ofthe invention.

FIG. 1 is a context diagram of a network communications environmentdepicting pseudowires interconnecting PE routers across a core network;

FIG. 2 is a flowchart of packet loss detection in the network of FIG. 1;

FIG. 3 is an example of applying packet loss detection to a sequence ofpackets in the network in FIG. 1; and

FIGS. 4-6 are a flowchart in further detail of packet loss tracking of asequence of messages packets.

DETAILED DESCRIPTION

Configurations herein are based, in part, on the observation thatconventional pseudowire traffic monitoring does not process or diagnosethe aggregate counts from both endpoints of a particular pseudowire. Itwould be beneficial for the PE routers to periodically post packetstatistics to the far-end (opposed) PE router to enable queries based onthe full packet drop/loss context of the pseudowire or other connection.Configurations discussed herein substantially overcome the aboveshortcomings of conventional packet traffic monitoring by periodicallyexchanging traffic packet counts to maintain an accurate diagnosis ofthe pseudowire health as observable from either endpoint. Further, theraw packet counts are analyzed to identify misrouted and lost packets,as both are considered to assess network health and congestion. Thepseudowire statistics are maintained for each pseudowire emanating froma particular edge router, providing a complete view of pseudowiretraffic affecting the edge router. Such statistics are beneficial forproblem detection, diagnosis, and for verification of throughputcriteria such as those expressed in Quality of Service (QOS) termsand/or SLAs.

Conventional packet counter mechanisms do not compute or detect thepacket loss on a pseudo-wire directly and completely. While packets maybe injected into a pseudowire and counted on the other side using anetwork analyzer, this approach is not typically desirable given thelarge number of pseudowires typically deployed between any two provideredge devices (PEs). Furthermore, such an approach does not typicallytake into account the injection and processing of control plane packetson these pseudowires.

Packet loss calculation may be achievable by employing native serviceOAM (Operations and Maintenance) techniques such as ATM (AsynchronousTransfer Mode). However, such solutions do not take into account all ofthe traffic traversing the pseudowire. They also run end-to-end (betweenclients of the access circuits attached to the PWs), which impedesdetection of the packet loss on the actual pseudowire portion of theconnection.

Conventional sequence numbers on pseudowire frames exist in bothMPLS-based and L2TPv3 pseudowire tunneling technologies. The mostcommonly referenced use for pseudowire sequencing, however, is to ensurein-order delivery of packets across a pseudowire. Configurations hereinalso employ the sequence numbers, but for OAM rather than an as an aideto the pseudowire emulation.

Conventional mechanisms, therefore, employ sequence numbers typicallyfor detecting congestion within a network. The method proposed herein,in contrast, further addresses troubleshooting and OAM, as well asnetwork congestion. Further, the method presented herein providesspecific user query ability, in addition to an active congestion controlmechanism using sequence numbers. Conventional counters maintain onlytransmit and receive counters, and do not propagate the counts to theopposed endpoint for identifying lost packets. Accordingly,configurations herein perform a count of packets that were neverreceived through a message count synchronization message, which posts(exchanges) message tracking statistics across both opposed ends(termination points) of a pseudowire or other suitable connection.

FIG. 1 is a context diagram of a network communications environmentdepicting pseudowires interconnecting PE routers across a core network.Referring to FIG. 1, a network communications environment 100 includes aservice provider core network 110, such as a packet switched networkoperable for message traffic transport using MPLS, IP, or L2TPV3. Thecore 110 includes a plurality of provider edge nodes PE-1 . . . PE-3(PE, generally), each interconnecting with one or more customer edge(CE) routers CE-1 . . . CE-4 (CE generally). The customer edge routersCE connect to a local subnet 120-1 . . . 120-4 (120 generally) such as aLAN, VPN, or other user interconnection. Pseudowires 150-1-150-3 (150,generally) connect a pair of provider edge PE routers for enablingtransport across the core network 110. Generally, the pseudowires 150each interconnect a pair of PE routers, and may include multiplepseudowires 150 between the same PEs 150-1 . . . 150-2. Accordingly, thepseudowires 150 collectively form an interconnection matrix through thecore 110. As indicated above, each of the pseudowires 150 emulates aparticular transport service, or mechanism, such as an LSP (LabeledSwitch Path) in an MPLS environment, for example.

FIG. 2 is a flowchart of packet loss detection in the network of FIG. 1.Referring to FIGS. 1 and 2, the method of tracking packet loss asdisclosed herein includes selectively identifying a routerinterconnection for packet anomaly (i.e. lost and misrouted) detection,as depicted at step 200. In the exemplary configuration, pseudowiresbetween PE routers are scrutinized, however alternate configurations mayexamine other interconnection mechanisms, as discussed further below.The mechanism, upon invocation, begins tracking packet anomalies of atleast one termination point of the router interconnection (i.e.pseudowire 150), as shown at step 201. Typically, both terminationpoints initiate tracking, or counting, of the packet sequence numbersreceived, thus enabling packet loss querying from either endpoint. Eachof the routers periodically exchanges packet anomaly statistics tosynchronize packet anomaly counts between the termination points of therouter interconnection, in which the anomaly statistics correspond tothe termination point and a particular opposed termination point, asshown at step 202. As each router maintains the raw counts of loss andmisroutes locally, a periodic exchange by each endpoint of thepseudowire enables a full pseudowire status from a query at eitherendpoint.

In a simplified service provider network scenario, a pseudowire might becharacterized as follows:

CE1-PE1<-pseudowire->PE2-CE2

PE1 and PE2 represent the provider equipment, and the pseudowireemulates (i.e.: carries) the service's traffic transparently between PE1and PE2 over a Packet Switched Network (PSN) core such as an MPLS or IP(L2TPv3) network 110. In order to overcome the above-describedshortcomings, configurations herein provide a mechanism that allows anend-to-end pseudowire 150 packet loss detection mechanism for alltraffic (data and control plane) that traverses the pseudowire. Notethat in the exemplary MPLS network shown, the pseudowire connects the PEnodes (routers) across the core, and does not extend to the customeredge routers, which are typically coupled by another routing mechanismsuch as IP, LSP or other suitable link. Alternatively, In a VirtualPrivate Wire Service (VPWS), the pseudowire effectively extends the wireconnection from the CE to the PE. In a Virtual Private LAN Service(VPLS), the pseudowires themselves are literally their own unique wiresand may not have a one-to-one correlation to the CE wires.

The specific techniques being proposed below employ the packetsequencing mechanisms described in L2TPv3 and MPLS pseudowires 150. Tounderstand this solution we first provide a bit of history on pseudowiresequencing. L2TPv3 provides a 24 bit sequence number and MPLSpseudowires provide a 16 bit sequence number. The typical behavior forsequencing is to ensure in-order delivery of packets received at PE1 andtransmitted across the PSN to PE2 and vice-versa. When an out-of-orderpacket is received at either PE1 or PE2 from the pseudowire, the typicalbehavior is to either drop the packet or to buffer the packet in anattempt to reorder it. Particular L2VPN implementations may provideout-of-order drop behavior; however do not attempt to reorder packets.As packets are dropped due to ordering problems, a counter isincremented. Most service providers do not enable sequencing today aspacket ordering issues typically do not cause problems in the protocolsin use today. Many IP applications are not sensitive to packet orderingissues. In addition, due to the way service providers engineer the PSNnetworks for providing L2 services, packet-ordering issues are usuallynot encountered. In any case, sequencing is infrequently desired oremployed in typical L2VPN/VPWS configurations.

In the configurations described herein, rather then employ sequencing inan attempt to ensure packet ordering, these configurations employsequencing to count packets which were never received. The pseudowiretermination points (e.g. PE routers) maintain two counters for eachpseudowire 150 in order to provide the statistics to infer thisinformation. The counters are:

1) Packets lost (134-3, below): Upon receiving a packet from thepseudowire, inspect the sequence number to determine whether the packetis in order, out-of-order (old), or in the future. If the packet is inthe future, then subtract the number of packets between the receivedsequence number and the expected sequence number and add this to arunning counter of packets lost, i.e.:

Expected sequence #: 5

Received sequence #: 10

Packets lost#: 5

2) Packets received out-of-order (134-4, below): This counter provides amethod to determine whether the packets missed counter is a result ofpackets which were dropped in the provider network or just reordered.This computation is modeled as followed:

Expected sequence # 11

Received sequence #: 7

Packets out-of-order++

In the examples above, we would have counted 5 packets lost and onepacket received out-of-order. From these 2 counters, we could infer thatwe lost a total of 4 packets.

By maintaining these two statistics and periodically extracting thesecounters from the data-plane into the control plane, the serviceprovider can gage the overall “health” of their pseudowires 150 in termsof packet loss. Any particular PE only knows half of the end-to-endhealth of the pseudowire 150. All that can be monitored directly on anygiven PE is the number of packets not received from the far end PE.Conventional mechanisms are unable to directly know whether the packetsa given PE is sending to a peer are actually making it there. Only thefar-end PE knows this information. Alternatively, configurations hereinpropose a mechanism where the PEs periodically post their statistics tothe far-end PE using an control channel message. This control channelmessage could take on a number of forms, i.e. the L2TPv3 Wan ErrorNotification message, a BFD keep-alive response packet transportedinside of the VCCV control channel, or a LSP status message. Byproviding each PE a periodic snapshot of their “packets lost” counters,both PEs have an overall idea of the health of the pseudowire 150 andcan report it as such to the operator. This is particularly important inthe case where both ends of the pseudowire reside within differentprovider networks (i.e.: inter-provider or intra-provider networks).

The service provider may set thresholds based on these packetdiagnostics such that when a certain packet loss threshold is reached,an alarm is generated and/or additional more detailed pseudowireanalysis is triggered. The interval at which this invention runs isconfigurable by the operator, and is configurable on both ends of thepseudo-wire.

Since inclusion of sequence numbers may be expensive in terms of packetprocessing, the service provider may not want sequencing to be enabledall the time. Thus in some situations, it may be desirable to enablethis packet diagnostic service dynamically, (i.e. based on customerfeedback about problems in the service.) In order to enable this dynamicbehavior, particular configurations selectively negotiate sequencingduring pseudowire session establishment, and ensure that presence of thecontrol word (which contains the sequence number) in either L2TPv3 orMPLS PWs is negotiated up front. The control word, as described above,initiates the counts at the opposed end of the pseudowire, and alsodistinguishes control traffic from data payload traffic. By ensuringthat the control word is present, configurations may dynamically enablesequencing without having to re-establish the signaling and data-planeassociated with the pseudowire. The provider would just need to indicatethat “packet-loss detection” should be enabled on an up-and-runningpseudowire that would then trigger an out-of-band message, or in-bandmessage running on top of VCCV, to request that “packet-loss detectionbe enabled”. Based on this event, the behavior described above would bydynamically enabled. The same type of exchange would be used to disablethe packet loss algorithm. Furthermore, the configuration of a packetloss threshold can be used to trigger a back-up pseudowire (presumablyover a different network path or to different equipment) or other typeof connection to be utilized instead of the primary one. This thresholdcan be used in collaboration with a timer that can be used to measure aspecific packet loss over a certain period of time, or even an averagepacket loss over a period of time.

FIG. 3 is an example of applying the packet loss detection scenariodescribed above to a sequence of packets in the network in FIG. 1.Referring to FIG. 3, a router such as CE-6, operable according totechniques disclosed herein, employs a count repository 130, such as theexemplary packet history table, for maintaining the counts. Note thatthe repository 130 need not retain the entire packet history, as shownhere for exemplary purposes. The exemplary repository 130 includesentries 132-1 . . . 132-6 (132, generally) for each packet, and containsfields for packets expected 134-1, packets actually received 134-2, acumulative count of packets lost 134-3, and a cumulative count ofpackets misrouted 134-4. Particular configurations may maintain arunning count of the lost 134-3 and misrouted 134-4 packets, rather thanthe entire table shown. A router CE-5 sends a plurality of messages 1 .. . 10 to router CE-6, over the path including the pseudowire 150-4between PE-4 and PE-5.

As indicated by the repository 130 table, entries 132-1 . . . 132-4indicate that packets 1-4 were received in order, resulting in zerocounts for lost 134-3 and misrouted 134-4 packets. After receivingpacket 4, packet 5 is expected, however packet 10 arrives, as shown byentry 132-5. Accordingly, the lost 134-3 count is computed: 10-5=5packets lost. Packet 11 is now expected, as shown by entry 132-6,however packet 7 arrives. Since packet 7 precedes the expected packet11, the misrouted count is incremented. Therefore, the misrouted packetcount 134-3 mitigates the lost count 134-4 by accounting for packetsdeemed lost but later arriving. Further, packets 8 and 9 will contributeto the misrouted count 134-4, as will packets 5 and 6 if ultimatelyrerouted from PE-4. Therefore, analysis of the counts includes examiningboth counts, as packets deemed lost due to a misordering may ultimatelyarrive, resulting in a smaller difference between the lost 134-3 andmisrouted 134-4 counts.

FIGS. 4-6 are a flowchart in further detail of packet loss tracking of asequence of messages packets. In an environment such as that in FIGS. 1and 3, an exemplary configuration operates as follows. Packet losstracking and analysis includes selectively identifying a routerinterconnection for packet anomaly detection, as depicted at step 300.An operator or system administrator may enable and disable the trackingmechanism as desired. Such a router interconnection is defined by twotermination points of a pseudowire 150, as shown at step 301. Theoperation of tracking packet anomalies is initiated for at least onetermination point of the router interconnection, as disclosed at step302, however typically is enabled at both ends.

The tracking initiates counts at a termination point (PE router) of apseudowire 150, as shown at step 303. One of the PE routers beginstransmitting the counts in a control message to the opposed terminationpoint of the pseudowire, as depicted at step 304. The receiving routerrecognizes the control message as an indication to begin tracking, andcommences, responsive to the received control message, the counts at theopposed termination point, as shown at step 305. In the exemplaryconfiguration herein, the control messages are Virtual CircuitConnectivity Verification (VCCV) messages, distinguished by a controlbit, however any suitable messaging mechanism may be employed. Uponinitiation, each of the endpoint PE routers counts the received packets,typically by observing the sequence number. Accordingly, maintaining thecounts further includes writing a sequence number in each packet, inwhich the sequence number corresponds to a set of packets sent on aparticular pseudowire, as shown at step 306. The receiving router at theopposed termination point receives the sequence number in the packet atthe termination end of the pseudowire, as depicted at step 307.

The receiving PE denotes packets indicative of count information via thecontrol field discussed above operable to indicate control informationas depicted at step 308. A check is performed, at step 309, to determineif a particular received packet includes count information. If not,control reverts to step 308 for receipt of the next packet. Note that intypical messaging environment sequence numbers may be a typical field ofa packet. However, the control messages, such as the VCCV messages forinitiating and posting (exchanging) statistics to peer PE routers areadditional messages beyond the payload bearing traffic.

Upon receipt, the receiving PE router compares the sequence number toprevious sequence numbers to identify order and omission of packets, asshown at step 310. As shown in the discussion of FIG. 3 above, at aparticular termination point of a pseudowire 150, for each pseudowiretermination, the router inspects the packet to determine if the packetis in order, preceding a previously receive packet, or advanced from anext expected packet, as depicted at step 311. A check is performed, atstep 312, to determine if the sequence number is indicative of anout-of-order packet. If not, control reverts to step 308 for receipt ofthe next packet. Otherwise, a further check is performed, at step 313,to determine if the packet is advance, indicating possible loss ofintervening packets, or misordered, indicating a later sequence numberwas previously received. Mathematically, this amounts to determining ofthe received sequence number is greater or less than the expectedsequence number, as outlined above.

If the received sequence number is greater than the expected sequencenumber, then the intervening packets are presumed lost and the receivingrouter PE maintains the packet loss count accordingly, as shown at step314. Since the received packet is advanced, the router PE-2 computes thedifference of the received and expected packet numbers, and incrementsthe lost packets by the computed difference, as depicted at step 315.

If the received sequence number is less than the expected sequencenumber, than the receiving router maintains a count of misorderedpackets, as shown at step 316. Since the received packet sequence No.precedes the expected packet, the router PE-2 increments the out oforder packets, as shown at step 317. Note that purported lost packets,as accounted for by the advanced packets received, are mitigated by themisordered count when the “skipped:” packets are ultimately received asa lesser sequence number than the expected packets. One aspect to thedistinction of misordered vs. lost packets is that some applicationsmaybe restricted to receiving packets in the proper order and maytherefore need to reorder misordered packets accordingly.

For each particular pseudowire for which the counts are enabled, thepertinent routers (PE-5,PE-6) maintain counts at both terminationpoints, as depicted at step 318. In other words, the mechanism maintainscounters at pseudowire termination points, in which the pseudowire isdefined by a set of labeled paths between provider edge routers andoperable as a virtual circuit through the core network, as depicted atstep 319. In particular configurations, the pseudowires may beaggregated or concatenated, such as across multiple service providernetworks. Accordingly, intervening routers maintain counts at aplurality of successive endpoints, in which the successive endpoints areoperable for concatenation into a larger virtual circuit, such that eachof the pairs of successive endpoints defining a particular providersubnetwork operable by a particular independent provider, as disclosedat step 320. Therefore, in a large core network, tracking the countsinvolves maintaining the counts for a plurality of pseudowires, asdepicted at step 321. Depending on the configuration, a number ofpseudowires may terminate in a particular PE router, interconnected withother PE routers and/or aggregated with pseudowires in the networks ofadjacent service providers.

At each pseudowire, in order to provide the peer to peer posting ofstatistics that enables querying from either endpoint, maintaining thecounts further includes periodically exchanging packet anomalystatistics to synchronize packet anomaly counts between the terminationpoints of the router interconnection, in which the anomaly statisticscorresponding to the termination point and a particular opposedtermination point, as depicted at step 322. Therefore, exchanging packetanomaly statistics between PE endpoints of a particular pseudowire 150includes posting statistics to the remote PE router by at least on of aVCCV status, L2TP WAN error notification, BFD keep alive, and an LSPstatus message, as shown at step 323. Further, the exemplary pseudowireconnection may be an alternate connection, such as an IP flow, tunnel,or other virtual or physical arrangement characterized by endpoints andsequential packets.

Statistics are exchanged, or posted, to opposed PE routers by exchangingVCCV messages between each of the termination points (e.g. PE routers)to propagate consistent counts of at least lost packets and misorderedpackets, as depicted at step 324. Additional counts may be included inalternate configurations. Such reporting may further include aggregatingthe counts for each of the pseudowires 150, as shown at step 325, suchas in the case of a large number of pseudowires from which to query.Control then reverts to step 308 for additional message traffic packetsand further processing as discussed above.

Note that the methods disclosed herein are described with specificreference to a pseudowire implementation and the L2TPv3 and MPLS/PWE3protocols. However, the use of sequence numbers to determine packet lossfor OAM is applicable to a variety of situations. For example, IPsec,GRE and L2TPv2 (RFC2661) typically employ a sequence number operable ina similar manner (though GRE lacks a standard control channel feedbackmechanism). IPsec is particularly interesting given its ability tonegotiate specific IP filters to apply traffic to. The negotiatedfilters in IPsec are operable to apply to specific IP flows that onewished to analyze actual traffic loss on among points in a network.

Those skilled in the art should readily appreciate that the programs andmethods for tracking packet loss as defined herein are deliverable to aprocessing device in many forms, including but not limited to a)information permanently stored on non-writeable storage media such asROM devices, b) information alterably stored on writeable storage mediasuch as floppy disks, magnetic tapes, CDs, RAM devices, and othermagnetic and optical media, or c) information conveyed to a computerthrough communication media, for example using baseband signaling orbroadband signaling techniques, as in an electronic network such as theInternet or telephone modem lines. The operations and methods may beimplemented in a software executable object or as a set of instructionsembedded in a carrier wave. Alternatively, the operations and methodsdisclosed herein may be embodied in whole or in part using hardwarecomponents, such as Application Specific Integrated Circuits (ASICs),Field Programmable Gate Arrays (FPGAs), state machines, controllers orother hardware components or devices, or a combination of hardware,software, and firmware components.

While the system and method for tracking packet loss has beenparticularly shown and described with references to embodiments thereof,it will be understood by those skilled in the art that various changesin form and details may be made therein without departing from the scopeof the invention encompassed by the appended claims. Accordingly, thepresent invention is not intended to be limited except by the followingclaims.

1. A method of tracking packet loss comprising: selectively identifyinga router interconnection from a plurality of router interconnections forpacket anomaly count detection; tracking packet anomalies at atermination point of the router interconnection; and periodicallyexchanging statistics of packet anomaly count detection to synchronizepacket anomaly counts between the termination point of the routerinterconnection and a particular opposed termination point of the routerinterconnection, the anomaly statistics corresponding to the terminationpoint and a particular opposed termination point, wherein the routerinterconnection is defined by two termination points of the pseudowireincludes a pseudowire.
 2. The method of claim 1 wherein tracking packetanomalies further comprises: maintaining a packet loss count; andmaintaining a count of misordered packets.
 3. The method of claim 2wherein maintaining the counts further comprises: writing a sequencenumber in a packet, the sequence number corresponding to a set ofpackets sent on a particular pseudowire; receiving the sequence numberin the packet at the termination point of the pseudowire; and comparingthe sequence number to previous sequence numbers to identify order andomission of packets.
 4. The method of claim 3 wherein exchangingstatistics of packet anomaly count detection further comprises postingstatistics to a remote PE router that is a termination point by at leastone of a VCCV status, L2TP WAN error notification, BFD keep alive, andan LSP status message.
 5. The method of claim 1 further comprising:maintaining at least one count indicative of the packet anomalies atboth termination points; and exchanging connectivity status messagesbetween each of the termination points to propagate consistent counts oflost or misordered packets.
 6. The method of claim 5 further comprisingmaintaining the counts at a plurality of successive endpoints, thesuccessive endpoints operable for concatenation into a larger virtualcircuit, each of the pairs of successive endpoints defining a particularprovider subnetwork operable by a particular independent provider. 7.The method of claim 6 wherein propagating consistent counts furthercomprises: aggregating the counts for each of the pseudowires; anddenoting packets indicative of count information via a control fieldoperable to indicate control information.
 8. The method of claim 7wherein maintaining the counts comprises maintaining counters atpseudowire termination points, the pseudowire defined by a set oflabeled paths between provider edge routers and operable as a virtualcircuit through the core network.
 9. The method of claim 8 furthercomprising: initiating counts at a termination point of the pseudowire;transmitting the counts in a control message to the opposed terminationpoint of the pseudowire; and commencing, responsive to the receivedcontrol message, the counts at the opposed termination point.
 10. Themethod of claim 9 further comprising maintaining counts for a pluralityof pseudowires, maintaining further comprising: at the termination pointof the pseudowire, for each pseudowire termination, inspecting thepacket to determine if the packet is in order, preceding a previouslyreceive packet, or advanced from a next expected packet; if the packetis advanced, computing the difference of the received and expectedpacket numbers, and incrementing the lost packets by the computeddifference; and if the packet is preceding, incrementing the out oforder packets.
 11. A data communications device for tracking packet losscomprising: a processor; a memory responsive to the processor andoperable for storing and retrieving instructions for tracking packetloss; an interface coupled to a network for providing at least onerouter interconnection from a plurality of router interconnections toanother data communications device, the interface responsive to theprocessor for selectively identifying a router interconnection forpacket anomaly count detection; and at least one counter operable totrack packet anomalies at a termination point of the routerinterconnection, the instructions further operable to periodicallyexchange statistics of packet anomaly count detection to synchronizepacket anomaly counts between the termination points of the routerinterconnection, the statistics of packet anomaly count detectioncorresponding to the termination point and a particular opposedtermination point, wherein the router interconnection is defined by twotermination points of the pseudowire includes a pseudowire.
 12. The datacommunications device of claim 11 wherein the at least one counter isfurther operable to tracking packet anomalies by: maintaining a packetloss count; and maintaining a count of misordered packets.
 13. The datacommunications device of claim 12 wherein the data communications deviceis further operable to: write a sequence number in a packet, thesequence number corresponding to a set of packets sent on a particularpseudowire; receive the sequence number in the packet at the terminationend of the pseudowire; and compare the sequence number to previoussequence numbers to identify order and omission of packets.
 14. The datacommunications device of claim 13 wherein the data communication deviceis further operable to post statistics to a remote PE router by at leastone of a VCCV status, L2TP WAN error notification, BFD keep alive, andan LSP status message.
 15. The data communications device of claim 14wherein the instructions further operable to maintain counts at aplurality of successive endpoints, the successive endpoints operable forconcatenation into a larger virtual circuit, each of the pairs ofsuccessive endpoints defining a particular provider subnetwork operableby a particular independent provider.
 16. The data communications deviceof claim 15 wherein the at least one counter further comprises aplurality of counters at pseudowire termination points, the pseudowiredefined by a set of labeled paths between provider edge routers andoperable as a virtual circuit through the core network.
 17. The datacommunications device of claim 16 wherein the data communication deviceis further operable to: initiate counts at a termination point of thepseudowire; transmit the counts in a control message to the opposedtermination point of the pseudowire; and commence, responsive to thereceived control message, the counts at the opposed termination point.18. The data communications device of claim 17 wherein the datacommunication device is operable to maintain counts for a plurality ofpseudowires, maintaining further comprising: at the termination point ofthe pseudowire, for each pseudowire termination, inspecting the packetto determine if the packet is in order, preceding a previously receivepacket, or advanced from a next expected packet; if the packet isadvanced, computing the difference of the received and expected packetnumbers, and incrementing the lost packets by the computed difference;and if the packet is preceding, incrementing the out of order packets.19. A computer readable medium encoded with a computer program fortracking packet loss, the computer program product comprising: computerprogram code for selectively identifying a router interconnection from aplurality of router interconnections for packet anomaly count detection;computer program code for tracking packet anomalies at at least onetermination point of the router interconnection, tracking packetanomalies further comprising: at a particular termination point of apseudowire, for each pseudowire termination, inspecting the packet todetermine if the packet is in order, preceding a previously receivepacket, or advanced from a next expected packet; if the packet isadvanced, computing the difference of the received and expected packetnumbers, and incrementing the lost packets by the computed difference;and if the packet is preceding, incrementing the out of order packets;and computer program code for periodically exchanging statistics ofpacket anomaly count detection to synchronize packet anomaly countsbetween the termination points of the router interconnection, thestatistics of packet anomaly count detection corresponding to thetermination point and a particular opposed termination point, whereinthe router interconnection is defined by two termination points of thepseudowire.
 20. A data communications device for tracking packet losscomprising: means for selectively identifying a router interconnectionfrom a plurality of router interconnections for packet anomaly countdetection; means for tracking packet anomalies at at least onetermination point of the router interconnection; and means forperiodically exchanging statistics of packet anomaly count detection tosynchronize packet anomaly counts between the termination points of therouter interconnection, the statistics of packet anomaly count detectioncorresponding to the termination point and a particular opposedtermination point; means for writing a sequence number in a packet, thesequence number corresponding to a set of packets sent on a particularpseudowire; means for receiving the sequence number in the packet at thetermination end of the pseudowire, wherein the router interconnection isdefined by two termination points of the pseudowire; means for comparethe sequence number to previous sequence numbers to identify order andomission of packets; and means for posting statistics to a remote PErouter by at least one of a VCCV status, L2TP WAN error notification,BFD keep alive, and an LSP status message.